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Today, hundreds of studies on the technique known as Realistic Mathematics 
Education (RME) have been discovered. Numerous trials yielding contradicting 
results. The purpose of this meta-analysis is to ascertain the overall effect of RME 
and to examine the moderator variables in order to explore the consequences. The 
ERIC database, Sage publications, Springer publications, semantic scholars, and 
Google scholars were used to identify empirical data. The study examined 54 
effect sizes from 38 individual studies conducted in the past two decades, 
involving 6140 participants. The estimation procedure was according to the 
random effects model, and statistical calculations were performed using the CMA 
program. According to the research findings, the study's overall effect size was 
0.97. This suggests that implementing RME has a significant favourable influence 
on pupils’ mathematical abilities. Moderator variables analysis shows that RME 
implementation would be more efficient when considering sample size, 
intervention length, learning mix, and education level. This finding contributes to 
the future implementation of RME to consider those variables that are shown to 
moderate the studies' effect sizes. Finally, comparisons of RME across countries 
and limitations of the study are discussed, providing vital information as a starting 
point for later studies and development of RME. 


Keywords: realistic mathematics education, mathematical ability, meta-analysis, 
moderator variables, effect size 


INTRODUCTION 


Mathematical ability is ability to complete mathematical tasks and effectively solve 
given mathematical problems (Karsenty, 2014). Students' mathematical abilities can be 
formed using relevant learning approaches (Marzuki, Wahyudin, Cahya, & Juandi, 
2021; Nurjanah, Latif, Yuliardi, & Tamur, 2020). Same with that Giiler (2018) 
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highlighted the importance of students to connect with mathematics courses in real life. 
Meanwhile, the use of context as a starting point for learning can promote students' 
academic abilities and make mathematics more enjoyable (Laurens, Batlolona, 
Batlolona, & Leasa, 2018; Ozdemir, 2017; Sawatzki, 2015). 


Learning that makes this real situation the starting point of learning is a feature of the 
Realistic Mathematics Education (RME) approach. RME is a unique teaching theory for 
the mathematical domain (Van den Heuvel-Panhuizen & Drijvers, 2014). This 
approach's peculiarity is that it gives rich and "realistic" situations a prominent position 
in learning activities. This situation acts as an entry point to explore students' initial 
knowledge and gradually becomes more formal (Van den Heuvel-Panhuizen & Drijvers, 
2020). RME today inspired international curriculum and colored mathematics learning 
(Prahmana, Sagita, Hidayat, & Utami, 2020). RME helps teachers to improve students' 
math skills (Hasibuan, Saragih, & Amry, 2019; Tanujaya, Prahmana, & Mumu, 2017). 
This is because RME equips students with the tools necessary to think critically, discuss, 
and solve real-world problems. This recommendation generated a research frenzy on 
RME. 


The RME method is known to influence mathematical abilities of students. However, 
previous studies into this theoretical assumption produced mixed results. Research 
conducted by Ardiyani (2018), Hirza & Kusumah (2014), Palinussa (2013), Saleh et al. 
(2018), Son, Darhim, & Fatimah (2020) demonstrates the efficacy of the RME strategy. 
Meanwhile, some research results do not support these findings (Yuniati, Armiati, & 
Musdi, 2020; Yuanita, Zulnaidi, & Zakaria, 2018). Multiple investigations within a 
single study frequently produce inconsistent, and as a result, forming conclusions 
concerning research issues can be subjective (Franzen, 2020; Juandi & Tamur, 2021). 
Educators want precise information that can be considered when implementing RME. In 
other words, systematic study is required to comprehend the relationship between 
mathematical talents and the moderator's role in RME application. 


Quantitative studies must be included to close this gap and give useful data for 
policymakers (Higgins & Katsipataki, 2015). As a result, a meta-analysis research is 
required to synthesize and evaluate the findings of individual investigations, as well as 
to analyze the causes of result variance for consideration in their future application. 
Meta-analysis provides in-depth and accurate conclusions (Fadhli, Brick, Setyosari, 
Ulfa, & Kuswandi, 2020; Siddaway, Wood, & Hedges, 2019). Thus when definitive 
conclusions are desired, the results of numerous individual studies must be taken into 
account using meta-analysis methods (Franzen, 2020). 


Previously, Chen, Shih, & Law (2020), Clark, Tanner-Smith, & Killingsworth (2016), 
and Wouters & Van Oostendorp (2013) conducted a meta-analysis study on the effects 
of gamification approach related to RME. The study's findings showed that introducing 
kids to learning through games had a little impact on their mathematical achievement. 
Game-based learning is one of the characteristics of RME, but it does not mean that 
RME is equated with that method. As a result, the research findings do not adequately 
explain the effect of RME. 
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In previous study, we studied the effect of RME over the last ten years and found that 
applying RME has a reasonably strong positive effect on the mathematical abilities of 
students (Tamur, Juandi, & Adem, 2020). However, this study only includes individual 
studies on RME in Indonesia. In line with that, analysis of studies from abroad and 
within the geographical region is needed to conduct a more up-to-date meta-analysis 
study (Ciftgi & Yildiz, 2019; Demirel & Dagyar, 2016). Additionally, the majority of 
these studies focus exclusively on unpublished research outputs, such as master's theses, 
thesis, and doctorate dissertations. Meanwhile, conducting the meta-analysis using only 
unpublished publications, such as theses, master's theses, and theses introduces 
publication bias (Borenstein, Hedges, Higgins, & Rothstein, 2009). 


Recent research has categorized the RME research trend, namely, reviewing 110 articles 
collected from a database of ten accredited national journals in Indonesia over the past 
two decades. The researchers identified seven categories, including the predominance of 
published RME articles, RME research themes, mathematics topics, student skills, RME 
terminology, and research methodologies utilized (Prahmana, Sagita, Hidayat, & Utami, 
2020). A study on RME in the last two decades has been conducted by Zulkardi, Putri, 
& Wijaya (2020). In their review, the researchers explained how RME came to 
Indonesia, the decision of the Indonesian government to send six doctoral candidates to 
the Netherlands, the results of the RME development to the development of the PMRI 
Center of Excellence, and the establishment of a doctoral program in the PMRI field 
(Zulkardi, Putri, & Wijaya, 2020). Both studies did not use quantitative methods and 
statistical tests. In line with that, Prahmana, Sagita, Hidayat, & Utami (2020) suggest the 
need for further studies that are more quantitatively oriented to arrive at more general 
results. 


This study expands previous studies by evaluating RME's overall impact on 
mathematical abilities of students. Investigations into the causes of effect size variation 
between individual studies were carried out by analyzing the relationship between 
identified moderator variables, namely, "difference in sample size," "difference in 
educational level," "duration of experimental intervention," "combination of learning," 
and "RME implementation countries." The results of this study provide educators with 
reliable information on future RME implementation. These ideas and the following 
questions drive this research. Does RME implementation produce larger effect sizes 
than conventional mathematical approaches? Do differences in sample size, level of 
education, experimental duration, learning combination, and geographical region of 
implementation of RME affect the study's overall effect size? 


METHOD 


This study employs meta-analysis to comb through a large number of papers in national 
and international journals. The study concerned the impact of applying RME to the 
mathematical domain. Generally, Borenstein et al. (2009) summarize the steps of meta- 
analysis, beginning with the establishment of inclusion criteria for the research under 
consideration. Second, methods for collecting empirical data and coding study variables 
are defined. Third, explaining statistic techniques.This work has also followed these 
steps. 
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Inclusion Criteria 


All papers included in the baseline search were screened and evaluated for consideration 
for inclusion in the meta-analysis based on the following criteria: (a) written in English; 
(b) peer-reviewed publication; (c) in Primary, Secondary Education Setting and higher 
education; (d) using the RME Approach; (d) the experimental study used a control class; 
(e) it contains sufficient statistical data for determining effect sizes, and (f) full article 
available. 


Data collection 


Studi utama diperoleh dengan menggunakan istilah yang relevan dari database online 
(see Figure 1). Data filtering using Preferred Reporting Items for Systematic Review and 
Meta Analysis (PRISMA; see Figure 1) protocol. The PRISMA protocol is a systematic 
review method that supports high-quality meta-analyzes (Pigott & Polanin, 2020). 


According to the search results, there were 241 studies that examined RME in total. 189 
studies passed the screening stage, but 103 did not match the criteria. Following that, 48 
studies were removed from the analysis due to the lack of complete statistical data. As a 
result, 38 distinct studies were included in the analysis. Due to the fact that several 
studies included more than one group, 54 independent samples were analyzed in this 
meta-analysis. The specific studies that were analyzed are included in Appendix 1. 


Records retrieved using databases Keywords used to search titels and abstracts 
(ERIC, Sage Publishing, Springer included: {realistic mathematics education} or 
Publishing, Scopus Database, and {realistic mathematics education-students' 
Google Scholar) mathematical abilities}, or {instructing realistic 

(n= 241) mathematics education approaches} and 


{realistic mathematics education effectiveness}. 
Duplicates records are removed 
(n= 48) 
Records excluded 


Record screened (n= 103 ) 
(n=189) Reason for exclusion: Written not in English, 
Inclusion Criteria inaccessible full text, Design studies, 
Qualitative studies, and development 


i 
3 
I 


Full-text articles assessed for 
eligibility 
(n= 86) 


Full-text articles excluded, 
With reasons 
(n= 48) 
Reason for exclusion: Descriptive statistical data 


as . 5 for effect size transformations are insufficient 
Studies included in a meta-analysis 
(n= 38) 


Figure 1 
A flow chart illustrating how PRISMA was used to conduct a meta-analysis of an 
experimental investigation on RME. 


Process of Coding and Reliability Test 


The studies that were included in the analysis were further coded according to the 
research objectives. A research instrument is a coding technique that is used to 
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transform the results of particular investigations into numerical data. This information 
includes sample size, differences in education levels, duration of experimental 
intervention, learning combinations, and countries of implementation of the RME ". By 
using this sheet, variable and effect size information were coded separately for each 
study by two researchers outside the project. Both are doctorate candidates who have 
completed courses in data analysis and meta-analysis in the past. In order to assess 
reliability, the Cohen's Cappa (k (7)) coefficient is used, this is an impressive statistic 
that is useful for the purpose of determining the degree of agreement between coders 
(McHugh, 2012). The calculation of Cohen's kappa is done according to the following 
formula: 

x(7) = Pria) — Pr{e) 

1—Pr(e) 


Where Pr (a) denotes an agreement that was actually seen and Pr (e) denotes a 
coincidental agreement. A deal level of 0.85 or greater is predefined to be considered 
high . An agreement level of 0.98 was obtained for this study. Thus there is a substantial 
to the almost perfect match between coders. 


Moderator Variables 


Meta-analysts always identify a moderating variable, that is, the characteristics of the 
individual study that are related to the study results (Hall & Rosenthal, 1991). The 
moderator in this study is a variable that has an effect on the effect of RME. The results 
of variable coding provide five identifiable moderators, namely: sample size, 
educational stage, duration of treatment, a combination of learning, and geographical 
region. The details of the five moderators are written in Table 1. 


Table 1 
Information about moderator variables 
Category Group N 
Sample Size 30 or less 23 
31 or over 31 
Primary school (PS) 17 
: Junior high school (JHS) 32 
Eguvaunnal elias Senior high school (SHS) 2 
College 3 
3-4 meetings 20 
Duration of treatment 5 meetings or more 23 
Unspecified 11 
Only RME 32 
Combination of learning RME + Math Software 11 
RME + Another method 11 
Indonesia 43 
Malaysia 1 
geographical region Atalland 2 
Turkey 2 
Greece 3 
United States of America 3 


International Journal of Instruction, January 2022 @ Vol.15, No.1 


386 A Meta-Analysis of the Last Two Decades of Realistic ... 


Statistic Analysis 


Analyzed data in the meta-analysis is the effect size (Glass, 2015). The effect size in this 
study is an index that quantifies the magnitude of the RME effect. Each analysis was 
calculated using the CMA software. The Cohen equation allows a fair approximation, 
but appears to overestimate the population size (Harwell, 2020). Use the Hedges' g 
equation to correct this overestimation. As a result, the effect size is calculated using 
Hedges' g equation, while the interpretation is based on Cohen's (1988) classification: 
less than 0.2 (negligible effect), between 0.2 and 0.5 (small effect), between 0.5 and 0.8 
(moderate effect), between 0.8 and 1.3 (large effect), and greater than 1.3 (large effect) 
(very large effect). The estimation approach use a random-effects model since it does 
not presuppose that all studies analyzed would yield the same genuine impact estimate 
(Pigott & Polanin, 2020). 


The p-value is checked, and CMA is used to calculate the heterogeneity test results. 
That all studies are different and therefore homogenous (Ho) is not accepted if the p 
value is less than 0.05. not possible to calculate the same population parameters 
(Borenstein et al., 2009; Kul, Celik, & Aksu, 2018; Juandi et al., 2021; Suparman, 
Juandi, & Tamur, 2021). In other words, the effect sizes between studies or categories is 
different (Turgut & Turgut, 2018). The heterogeneity between study groups suggests 
that differences in moderators affect the studies' effect sizes (Susanti, Juandi, & Tamur, 
2020; Tamur, Jehadus, Nendi, Mandur, & Murni, 2020; Yunita, Juandi, Tamur, Adem, 
& Pereira, 2020). Since the number of studies included in the analysis was relatively 
large, we adopted 5% (p <0.05) as the level of significance. 


A publication bias check was conducted to prevent misrepresentation of the findings. To 
the extent that any of the 38 studies included in the review had sample bias, it would 
show up in the overall summary of reported effect sizes (Borenstein et al., 2009). As a 
result of this publishing bias, academics are less inclined to publicize inconsequential 
discoveries than are statistically significant ones (6 percent) (Cooper, 2017). We are 
concerned that the results of this study may overstate the true magnitude of this effect 
(Arik & Yilmaz, 2020; Ferguson & Heene, 2012; Park & Hong, 2016). In order to 
anticipate this, funnel plots and Rosenthal's FSN statistics were examined to assess the 
possible amount of bias (Borenstein et al., 2009; Tamur, Juandi, & Kusumah, 2020); 
Paloloang, Juandi, Tamur, Paloloang, & Adem, 2020). This research is said to be bias- 
resistant if it reveals a symmetrical distribution along the vertical line (Borenstein et al., 
2009). If the impact size is not symmetrically distributed, Rosenthal's fail-safe N (FSN) 
statistic is used. If FSN/(Sk+10)>1, where k is the number of studies included in the 
meta-analysis, this research is immune to publishing bias (Mullen, Muellerleile, & 
Bryant, 2001). 


FINDINGS 


Data screening procedures were followed to ensure the studies used in this meta-analysis 
met the inclusion criteria. After this screening, 38 individual studies, providing 54 
independent samples, were analyzed. Table 2 presents the study's name, year of 
publication, N (sample size), Educational stage, and ES (effect size). 
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Table 2 
Summary of analyzed individual studies 


387 


No Study name Publication year N Educational stage ES 
1 Huntley et al., 2000a 2000 593 _ Junior high school 0,46 
2 Huntley et al., 2000b 2000 592 _ Junior high school 0,44 
3 Huntley et al., 2000c 2000 593 _ Junior high school 0,36 
4 Fauzan 2002, a 2002 74 Primary school 0,40 
5 Fauzan 2002, b 2002 70 Primary school 1,36 
6 Fauzan 2002, c 2002 69 Primary school 0,32 
fi Palinussa, 2013a 2013 47 Junior high school 0,57 
8 Palinussa, 2013b 2013 59 Junior high school 0,70 
9 Zaranis et al., 2013a 2013 62 College 0,12 
0 Zaranis et al., 2013b 2013 63 College 0,63 
1 Zaranis et al., 2013c 2013 64 College 0,43 
ei Susanti et al., 2014a 2014 185 Junior high school 0,54 
3 Susanti et al., 2014b 2014 53 Junior high school 0,46 
4 Susanti et al., 2014c 2014 132. Junior high school 0,42 
5 Hirza et al., 2014 2014 164 Primary school 0,85 
6 Zubainur et al., 2014 2014 50 Primary school 0,84 
a Wardono et al., 2016a 2016 37 Junior high school 1,16 
8 Wardono et al., 2016b 2016 55 Junior high school 1,78 
9 Mahendra, 2017 2017 63 Junior high school 1,47 
20 Habsah, 2017a 2017 59 Junior high school 2,07 
21 Habsah, 2017b 2017 59 Junior high school 1,90 
22 Karaca et al., 2017 2017 45 Primary school 1,21 
23 Supandi et al., 2017 2017 72 Junior high school 0,68 
24 Zakaria et al., 2017 2017 61 Senior high school 0,96 
25 Sumirattana et al., 2017a 2017 104 Junior high school 1,01 
26 Sumirattana et al., 2017b 2017 104 Junior high school 2,02 
27 Yuanita, 2018a 2018 426 Junior high school -0,36 
28 Yuanita, 2018b 2018 426 Junior high school 1,00 
29 Laurens et al, 2018 2018 50 Junior high school 2,72 
30 Altiparmak et al., 2018 2018 82 Primary school 11:7 
31 Septriyana et al., 2018 2018 40 Primary school 1,79 
32 Sofiyah et al., 2018 2018 66 Primary school 2,61 
33 Ramdhani et al., 2018 2018 60 Junior high school 1,37 
34 Laurens, 2018 2018 46 Primary school 0,71 
35 Warsito et al., 2018 2018 72 Junior high school 1,41 
36 Suryani., 2018 2018 42 Junior high school 0,49 
34 Laurens, 2018 2018 46 Primary school 0,71 
37. Kusumaningsih., 2018 2018 66 Junior high school 2,81 
38 Febriyanti., 2019 2019 50 Primary school 1,80 
39 Hasbi et al., 2019 2019 64 Junior high school 1,92 
40 Ndiung, 2019a 2019 102 Primary school 0,82 
41 Ndiung, 2019b 2019 100 Primary school 0,03 
42 Pertiwi, 2019 2019 69 Junior high school 1,09 
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Table 2 

Continued 
No Study name Publication year N Educational stage ES 
43 Amrina, 2019 2019 41 Senior high school 0,94 
44 = Junaedi, 2019 2019 78 Junior high school 0,40 
45 Umbara, 2019 2019 65  Juniorhigh school 0,82 
46 Ndiung et al., 2019a 2019 101. Primary school 0,04 
47 Ndiung et al., 2019b 2019 101 Primary school 0,86 
48  Marpaung et al., 2020 2020 66 Junior high school 0,62 
49 Dwietal., 2020 2020 58 Primary school 0,66 
50 ~=Sonet al., 2020 2020 95 Junior high school 1,41 
51 Yerizon, 2020 2020 64 Junior high school 1,92 
52 Kurino, 2020 2020 40 Primary school 1,32 
53 Yuniati, 2020a 2020 61 Junior high school 0,14 
54 Yuniati, 2020b 2020 60 Junior high school 1,43 


Note: To represent studies that produce more than one effect size, we use a, b, c. The effect size 
refers to the Hedges equation, g. 


When Table 2 was investigated, only 11.11% of individual studies were conducted 
before 2010, and 88.89% of individual studies were conducted between 2010 and 2020. 
In education, junior high school participant occupy the largest proportion (59%), 31% 
are students. PS, 4% are SHS students, and 6% are students. Furthermore, based on the 
main study countries conducted, Indonesia gave the largest contribution (78%), and the 
rest were scattered in Malaysia, Thailand, Turkey, Greece, and the United States. 
Individual studies with more than 30 students are 57%, and less than 31 students are 
43%. Based on treatment duration, 37% of individual studies were conducted over 3-4 
meetings, 42% of individual studies were conducted in a range of 5 or more meetings, 
and the rest were not determined. Finally, by considering the combination of learning, 
the study that only applies RME is 59%, followed by RME and mathematics software 
(21%), and the rest is RME with other methods. 


Results of the Overall Analysis to Address the First Question 


First, this study examined the overall impact of RME on the mathematical abilities of 
students. Figure 2 presents the research forest plots that illustrate the effect sizes, 
confidence intervals, and standard errors of the 54 studies. 
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‘Study name Statistics for each study Std diff in means and 95% Ci 


Std diff Standard Lower Upper 
in means error Variance limit limit ZValue p-Value 


Huntley et al., 2000a 0,457 0,083 0,007 0,294 5,478 0,000 
Huntley et al., 2000b 0,441 0,083 0,007 0,277 5,280 0,000 
Huntley et al., 2000c 0,362 0,083 0,007 0,200 4,362 0,000 
Fauzan 2002, a 0,401 0,236 0,056 -0,061 1,700 0,089 
Fauzan 2002, b 1,360 0,266 0,071 0,839 5,120 0,000 
Fauzan 2002, c 0,320 0,243 0,059 -0,155 1,321 0,187 
Palinussa, 2013a 0,298 0,089 -0,013 1,915 0,055 
Palinussa, 2013b 0,268 0,072 0,176 : 2,617 0,009 
Zaranis et al., 2013a 0,261 0,068 0,388 0,473 0,636 
Zaranis et al., 2013b 0,270 0,073 0,107 2,356 0,018 
Zaranis et al., 2013c 0,272 0,074 0,100 1,592 0,114 
Susanti et al., 2014a 0,150 0,022 0,252 3,639 0,000 
Susanti et al., 2014b 0,278 0,078 -0,080 1,672 0,095 
Susanti et al., 2014 0,177 0,031 0,078 2,401 0,016 
Hira et al., 2014 0,163 0,027 0,535 5,237 0,000 
Zubainur et al., 2014 0,295 0,087 0,263 2,851 0,004 
Wardono et al., 2016a 0,287 0,082 0,604 4,069 0,000 
Wardono et al., 2016b 0,319 0,102 1,158 z 5,593 0,000 
Mahendra, 2017 0,284 0,081 0,918 , 5,189 0,000 
Habsah, 2017a 0,323 0,104 1,439 6,418 0,000 
Habsah, 2017b , 0,314 0,098 1,287 , 6,062 0,000 
Karaca et al., 2017 i 0,324 0,105 0,578 3,740 0,000 
Supandi et al., 2017 0,242 0,059 0,206 2,808 0,005 
Zalaria et al., 2017 0,271 0,073 0,438 F 3,580 0,000 
Sumirattana et al., 2017a 0,208 0,043 0,609 4,884 0,000 
Sumirattana et al., 2017b , 0,241 0,058 1,551 z 8,393 0,000 
Yuanita, 2018a 0,098 0,010 -0,556 3,727 0,000 
Yuanita, 2018b 0,103 0,011 0,802 9,763 0,000 
Laurens et al, 2018 : 0,393 0,154 1,955 6,937 0,000 
Altiparmak et al., 2018 0,239 0,057 0,704 4,903 0,000 
Septriyana et al., 2018 0,375 0,140 1,063 , 4,796 0,000 
Sofiyah et al., 2018 A 0,335 0,412 1,956 5. 7,795 0,000 
Ramdhani et al., 2018 0,287 0,082 = 0,811 4,786 0,000 
Laurens, 2018 0,305 0,093 0,115 : 2,337 0,019 
Warsto et al., 2018 0,263 0,069 0,893 i 5,352 0,000 
Suryani., 2018 0,313 0,098 -0,122 1,572 0,116 
Kusumaningsh., 2018 , 0,348 0,121 2,133 8,088 0,000 
Febriyanti., 2019 0,340 0416 1,133 5,291 0,000 
Hasbi et al., 2019 0,305 0,093 1,325 , 6,306 0,000 
Ndiung, 2019a 0,206 0,043 0,419 ‘i 3,991 0,000 
Ndiung, 2019b 0,200 0,040 -0,357 0,173 0,863 
Pertiw, 2019 0,258 0,067 0,584 F 4,223 0,000 
Amrina, 2019 0,330 0,109 0,299 2,864 0,004 
Junaedi, 2019 0,229 0,052 -0,044 1,767 0,077 
Umbara, 2019 0,259 0,067 0,324 : 3,211 0,001 
Ndiung et al., 2019a 0,199 0,040 -0,355 0,178 0,859 
Ndiung et al., 2019b 0,208 0,043 0,461 ‘ 4,173 0,000 
Marpaung et al., 2020 0,252 0,064 0,126 2,457 0,014 
Dw et al., 2020 0,270 0,073 0,135 2,462 0,014 
Son et al., 2020 0,230 0,053 0,968 6,176 0,000 
Yerizon, 2020 0,302 0,091 1,334 4 6,369 0,000 
Kurino, 2020 : 0,349 0,122 0,638 3,789 0,000 
Yuniati, 2020a 0,256 0,066 -0,359 0,559 0,576 
Yuniati, 2020b 0,290 0,084 0,868 , 4,957 0,000 

0,086 0,007 0,817 11,450 0,000 


-1,00 0,50 


Favours A 


Figure 2 
Plot forest research 


WiMie tx ig sexayni it appears that there are wide levels of confidence and 
in AA: | esting clear heterogeneity in this study. Figure 2 also 
presents the random-effect model estimation analysis from 54 studies, and graphically 
illustrates the effect size (square point). The confidence interval estimates are a 
horizontal line extending from both sides. It appears that only one effect size is negative, 
while the other is positive. Furthermore, it is clear that the majority of effect sizes have 
narrow confidence intervals (57%), while 43% of effect sizes have slightly wide 
confidence intervals. Table 3 shows the overall effect sizes and the confidence intervals 
of these studies according to the meta-analysis method. 
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Tabel 3 

Definition of meta-analysis outcomes by estimation model 

Model N Hedges’s g Standard 95% Confidence Interval Q P Decision 
error Lower Upper 

Fixed-effects 54 0.67 0.02 0.62 0.72 487.19 0.00 Reject 

Random-effects 54 0.97 0.08 0.80 1.14 Ho 


Table 3 shows that effect size distribution is diverse. As a result, this study's meta- 
analysis procedure follows the random effect model exactly. Furthermore, to test 
whether there is publication bias or not, a funnel diagram can be taken into account. 
Figure 3 presents the funnel plot obtained in the study. 


Funnel Plot of Standard Error by Hedges's g 
0.0 . 


0.1 


0.2 


Standard Error 


0.3 


0.4 


Figure 3 
Research funnel plot 


As you can see in Figure 3, the distribution of effect sizes is not perfectly symmetrical. It 


was decided to analyze any publication bias using statistics from Rosenthal's fail-safe N 
(FSN). Table 4 summarizes the N test calculation findings. 


Table 4 
FSN statistical calculation results 
Bias Condition 


The z value for observed studies 28.61 
The P-value for observed studies 0.00 
Alpha 0.05 
Tails 2 
The z value for Alpha 1.96 
Number of Observed Studies 54 
FSN 1459 


From data analysis results, using CMA software, the Rosenthal safe N value is 1459. 
Based on the formula Mullen et al. (2001) then 1459 / (5 * 54 + 10), the calculation 
result is 4.56. It was discovered that this number was higher than 1. According to this 
estimation, the research examined were not skewed by the publishing of their findings. 
Second-Question Analysis: Results of Moderator Variables Analysis 


In light of the mixed findings in the meta-analysis, it is necessary to analyze the 
Moderator Variable, which is thought to have an impact on how dependent and 
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independent variables relate to one another (Arik & Yilmaz, 2020). The 54 effect sizes 
derived from 38 separate studies were analyzed in relation to moderator variables such 
as sample size, differences in educational attainment, duration of experimental 
intervention, learning combination, and geographical region of RME implementation. 
Table 5 summarizes the findings of the study. 


Table 5 
Summary of the results of moderator variable analysis 


F Heterogeneity Decision 
Category Group N_ Hedge's g (Qs) dhQ)P 
: 30 or less 23 1.19 : 
Sample Size BT hOvER 310.56 84.51 1 0.00 Reject Ho 
Primary school (PS) 17__0.80 
Educational Junior high school (JHS) 32 0.85 . 
stage Senior high school (SHS) 2 0.94 ne OS Gor. ‘Reect Ho 
College 3 0.38 
uence 3-4 meetings 20 (0.74 
eatien 5 meetings or more 23. 0.50 40.52 2 0.00 Reject Ho 
Unspecified 11 1.01 
cial Only RME 32. (0.75 
eae RME + Math Software 11 (0.96 854 2 0.01 Reject Ho 
RME+ Another method 11 0.58 
Indonesia 43 1.07 
Malaysia 1 0.98 
Geographical — Thailand 2 1.437 : 
bata : Turkey 2 1.172 TINO 5 O00) Baler 
Greece 3 0.40 
United States of America 3 0.42 


Table 5 displays the moderator's findings, including the five categories that were 
examined. First, the size of the sample provides two groups, namely 30 or less and 31 or 
over. As a result of the heterogeneity test, there was a significant difference between the 
two groups (P <0.05). Second, the educational stage category provided three groups, 
and the effect sizes of the three differed significantly (P <0.05). Third, the treatment 
category's duration provides t categories, and all three are significantly different (P 
<0.05). Fourth, the combination of learning gave three groups, and they were 
significantly different (P <0.05). Finally, based on the geographical area category, the 
country where the research was conducted gave different results for the six categories (P 
<0.05). This means that the five categories moderate the effect of RME on mathematical 
abilities. 


DISCUSSION 


A search of individual studies on RME was conducted using databases that include 
ERIC, Sage Publishing, Springer Publishing, Scopus, and Google Scholar. Study 
selection followed the PRISMA protocol. This stage identified 241 individual studies on 
RME, and only 38 studies were eligible for analysis. Design, qualitative, and 
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development research predominate in individual RME investigations. For the past two 
decades, design research has dominated research on RME in Indonesia, as described by 
Prahmana et al. (2020). 


The resulting analysis (see Table 3) found that this study's overall effect size is 
measured at 0.97 based on the random-effect model, which suggests that RME 
implementation has a major positive effect on the mathematical abilities of students. 
This is achievable because to RME's ability to enable students to collaborate, discuss, 
think, and find solutions to real-world problems. Results suggest that students in the 
experimental group who were rated 15th were roughly equivalent to those in the control 
group who were ranked sixth. These findings are in line with those of Tamur, Juandi, 
and Adem (2020), who examined 72 studies on the impact of RME on Indonesian 
students' mathematical abilities and reported an overall effect size of 1.10. This finding 
is not much different from previous meta-analyses that a game-based learning model is 
effective on mathematics learning achievement (Chen, Shih, & Law, 2020; Clark, 
Tanner-Smith, & Killingsworth, 2016; Wouters & Van Oostendorp, 2013). Although the 
number and search for this analysis differed from those of previous studies, this study 
showed quite similar results, a fact that indicates the superiority of RME. 


It becomes out that the five moderator factors have an impact on the study's overall 
effect size. The summary of results illustrated in Table 5 shows a strong association 
between RME effectiveness and sample size. The combined effect size for the small 
sample group (30 or less) differs significantly from the combined effect size for the 
large sample group (31 or more). A small study group's effect on a small sample is 
stronger than that of a large sample, according to Tamur, Juandi, and Adem (2020) and 
Tamur and Juandi (2020). These results prompt educational practitioners to consider 
sample size in further RME implementation. 


Although the previous meta-analysis showed homogeneity of the overall effect sizes 
between the study groups carried out at PS, JHS, and SHS (e.g., Tamur, Juandi, & 
Adem, 2020; and Tamur & Juandi, 2020), this study revealed different results. 
Descriptively this study reports a combined effect size that is almost the same as that of 
studies conducted in primary and secondary schools but differs significantly from 
studies conducted at universities. Chen, Shih, & Law (2020) found the same result that 
the average effect size of the studies that have been conducted in universities (ES = 
0.14) was smaller than the studies conducted in primary (ES = 0.67) and secondary 
(0.43). These findings provide a tentative assumption that RME is not recommended to 
be implemented at the university level. This is possible because in the RME framework, 
students are offered problem situations that they can imagine (Van den Heuvel- 
Panhuizen & Drijvers, 2014). Meanwhile, students studying at universities may no 
longer need this framework. In order to verify this allegation, further research is needed. 


There was a significant difference in effect size between groups that received therapy for 
three to four meetings and groups that received treatment for four or more sessions, as 
shown in Table 5. In the meantime, the effect magnitude was greater in the group whose 
treatment duration was not specified. The application of RME is most effective if the 
duration of the treatment is limited to four sessions. In studies in which the treatment 
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was continued for more than four meetings, the effect was reduced. These findings 
support the results of the research of Tamur, Juandi, & Adem (2020), which reported 
that the application of RME would get more done with less effort if the duration of 
treatment was not prolonged too much. The Hawthorne effect's higher effect size 
associated with shorter treatment duration can be explained (Bayraktar, 2001; Juandi, 
Kusumah, Tamur, Perbowo, & Wijaya, 2021; Tamur, Kusumah, et al. 2021). 
Hawthorne, or novelty, is an effect that occurs when students are stimulated to try harder 
simply because of the novelty of treatment. However, if the treatment is given for a long 
time, then its appeal will be lost. This finding is tentative because 11 individual studies 
did not inform the duration of treatment. 


The purpose of this study is to examine the effect of a combination of learning on the 
implementation of RME. The analytical results indicate that when RME is used in 
conjunction with mathematical software, it is advantageous. This study is consistent with 
the findings of a meta-analysis on the use of mathematics software in Indonesia, which 
concluded that the overall effect size is 1.162 (Tamur, Juandi, & Kusumah, 2020; 
Tamur, Fedi, et al., 2021; Tamur, Kusumah, et al., 2021). In this context, the 
employment of RME in conjunction with mathematical software helps to clarify the 
complexity of realistic modeling, and its application can allow in-depth problem study, 
even at an early stage (Bray & Tangney, 2016; Budinski & Milinkovic, 2017; Dewolf, 
Van Dooren, & Verschaffel, 2017; Kim et al., 2020). 


Mathematical learning supported by the RME approach has been applied worldwide. 
This study also analyzes the application of RME in various countries. It appears that 
RME has achieved a high level of effectiveness when implemented in Indonesia, 
Malaysia, Turkey, and Thailand. Meanwhile, the RME applied in Greece and the United 
States has little effect. This discrepancy in results could be explained by cultural or other 
factors that have not been identified. Another surprising point relates to the fact that the 
Netherlands was the first to develop RME but did not report any experimental studies on 
RME. In the Netherlands, the implementation of RME appears to be more oriented 
around design research. This is reflected in the statement by Van den Heuvel-Panhuizen 
& Drijvers (2020) that teachers must take a proactive part in building RME learning 
scenarios and designs. 


CONCLUSION 


Investigations have been undertaken to integrate findings from the impact of 
implementing RME over the past two decades. Several of the study's conclusions are 
that implementing RME has a significant positive effect on students' mathematical 
abilities when compared to traditional learning. RME is most effective when sample 
sizes are less than or equal to 30, treatment sessions are 3-4, and mathematical software 
is used. The application of RME is recommended in elementary schools, junior high 
schools, high schools, and not recommended in universities. RME is suitable to be 
implemented in Indonesia, Malaysia, Turkey, and Thailand compared to the United 
States and Greece. 
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The results of this analysis are only based on studies that meet the eligibility criteria. 
However, there have been found many related studies that cannot be analyzed due to 
limited statistical data. This study has not identified mathematical ability as a mediating 
variable. Besides, a study search strategy that relies solely on online has limited this 
study to analyzing only 38 studies from six countries. A manual search by visiting the 
library is necessary to reach unpublished publications such as theses and dissertations. 
Collaborative study amongst countries on the influence of RME is required in the future 
to produce more complete results. 
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